The invention relates to a user interface (UI), in particular to a multi-modal UI for devices with small screen size, such as handheld information appliances (palmtops, electronic watches, organizers, mobile phones, Web pads, remote controls, PDA""s or notebook computers, etc.)
Current trends indicate that by 2002 there will be 1B subscribers worldwide to mobile phones. By 2004 there will be more mobile phones in use than PC""s.
Mobile phones have become personal, trusted devices. Mobile phones, such as the Genie manufactured by Philips Electronics, typically have an LCD as part of the user interface and there is a trend to provide an even larger number of onboard software services. The phone has speech control capabilities as well.
Hand-held computing devices, such as personal digital assistants (PDA), e.g., the Palm manufactured by 3COM or a Windows CE-based iPAQ handheld, can be connected to the Network via a wireless modem, or 2.5 G (e.g., GPRS) or 3 G (e.g., DoCoMo) services. As a result, ubiquitous information access via the Internet has become reality. The latter services enable the device to stay always connected and to access information, such as e-mail, instant messages, phone calls over IP, audio/video and personal schedule or calendar. Wireless LAN and PAN connectivity standards, such as 802.11b and Bluetooth, enable PDAs and mobile phones to interact with a variety of information content, located on the respective networks. A combination of GPRS and 802.11b connectivity provides for seamless roaming between different networks: wide-area network (WAN), local-area network (LAN) and personal-area network (PAN). Studies further indicate that the functionalities of PDA""s and mobile phones have started to converge, and that a mobile information society is developing. There will be an emerging of dedicated devices. PDA""s are now work-related. In the near future, PDA""s will be personalized computers that stay with the user all the time. PDA""s will get more power and smaller size and accommodate more, and more versatile, functionalities. Most connected devices can be controlled via a UI by selecting an icon, a menu item or a list item using a touch screen, or by voice, by selecting an item using voice recognition software and hardware, or by using dedicated hardware interfaces, such as buttons, scroll wheels, etc.
Convenience and user-friendliness are believed to be the factors that limit the usability and practicality, be it a mobile phone, a palmtop, a PDA or an electronic watch. In particular, the UI and the service accessibility for connected personal devices have become critical factors for market acceptance. In particular, e-commerce or electronic shopping may benefit from the ubiquity of personal communication devices if the user interface technology enables users to employ best options for interaction with a wide variety of digital content and communication options.
This invention addresses, among other things, providing a method and apparatus to facilitate user-access to information, control and communication options by providing the user with multiple interface modalities, each of which is most suitable for a particular multi-step task. A further aspect of this invention relates to a method and apparatus to enable modality selection based on the information resolution for a particular information access, control or communication task. Another aspect of the invention enables the customization of the UI modality selection.
The term xe2x80x9cinformation resolutionxe2x80x9d used in this text refers specifically, but not exclusively, to a property of a collection of information items that determines the user""s ability to discriminate between different items when perceived via visual, auditory, tactile or other sensory means. For example, the resolution is associated with the relative distance between items selected or acted upon, when rendered on a display monitor (spatial distance), captured by a microphone or played out via a loudspeaker (temporal distance). The proper resolution or resolution interval may be dependent on the individual user, on the UI modality, device capabilities, environmental conditions and on other factors. For example, U.S. Pat. No. 6,211,856 issued to Sung Choi and Jan van Ee, and incorporated herein by reference, relates to a graphical user interface touch screen for displaying controllable functions of an electronic device. The function is displayed as an icon and at a scale size in which the function is recognizable by a user but too small to easily access features of the function. A magnified version of at least an area of the icon is provided upon the user touching the area of the icon. In the above-mentioned example, the first information resolution is too low for the user to affect the desired task, e.g. keystroke, with a high degree of confidence in its outcome. The magnified version increases the information resolution, specifically the control aspect of it, by increasing the size of each icon representing a corresponding action. In that case information resolution is increased directly in the same logical dimension (graphical representation) by increasing originally insufficient size to the acceptable level.
The inventor has realized that information resolution can be affected by using specific UI modalities to render the information for the user""s perception. Alternative UI modalities can be used to provide different ranges of information resolution. The term xe2x80x9cmodalityxe2x80x9d refers to one or more specific modes of sensation, or to one or more specific ones of the sensory perceptions (e.g., hearing, vision, kinesthesia) and/or corresponding physiological processes (e.g., voice input/output, auditory feedback, graphical input/feedback, touch input/feedback) involved in the user interaction with the UI. The information items can be represented explicitly (e.g., graphically, auditory) or implicitly (e.g., presence intuitively clear or by experience, or indicated by a mnemonic) in the UI.
Accordingly, the invention relates to an electronic device with a UI. The UI provides first user-selectable options. Second user selectable options available upon selection of a specific one of the first options. An information resolution of the first options, when rendered, differs from the information resolution of the second options when rendered. A first modality of user interaction with the UI for selecting from the first options differs from a second modality of user interaction with the UI for selecting from the second options. Preferably, at least the first modality or the second modality is programmable in a configuration mode of the UI. In this manner, the user can configure the way in which to interact with the UI depending on, e.g., his/her preference or the information resolution at the various levels in the menu. For example, in a multi-layered menu, a layer comprises a large number of first options, and another layer comprises a smaller number of second options. It may be more convenient to select from among the first options using, e.g., a touch screen, and from among the second options using voice input.
Preferably, an additional modality of the user interaction with the UI is available for the selecting from at least the first or the second options. In an embodiment of the invention, the device comprises a communication functionality, e.g., of a phone, an instant messager, or an email handler, etc.
In a further embodiment, the device has a touch screen, and a microphone for voice input. One of the first and second modalities involves touching the touch screen, and the other of the first and second modalities involves the voice input.
In another embodiment of the invention, the device comprises a display monitor for rendering at least some of the first options; first means for enabling the user to associate each respective one of the rendered first options with a respective one of a plurality of predetermined attributes; and second means for selecting a specific one of the rendered first options through the associated attribute.
The invention also relates to a method of providing a UI for enabling to select options arranged in first and second layers. When rendered in the UI, the options in the first layer have an information resolution that differs from the information solution of the options in the second layer when rendered in the UI. The method comprises providing a first modality to select a first one of the options in the first layer, and the first modality differs from a second modality to select a second one of the options in a second layer. The invention can be used with, e.g., a Web site to provide a user-friendly or customized interface.
The invention further relates to software for use on an electronic device for generating a UI that provides first user-selectable options, and second user selectable options available upon selection of a specific one of the first options, wherein an information resolution of the first options when rendered differs from the information resolution of the second options when rendered, and wherein a first modality of user interaction with the UI for selecting from the first options differs from a second modality of user interaction with the UI for selecting from the second options. An electronic device such as a laptop, PDA or mobile communication apparatus, thus can be upgraded to suit the needs and preferences of the user.